A Productive Method for Improving Test Effectiveness

Automated testing of software products has greatly expanded over the past few years. Ever increasing test suites have been developed, along with the computing infrastructure to support them. While the capacity for testing has grown, the environment is not infinitely scalable eventually capital spending is capped. Methodologies need to be explored that improve the overall effectiveness of the test cases that are run. Furthermore, these methodologies need to be as independent from the test suites as possible: the size of the test suites render solutions that are tightly bound to them ineffective for widespread utilization. Problem associated with these huge numbers of test cases is that whenever the code is changed, the entire suites of test cases need to be run. One idea is to run fewer tests on an ongoing basis, reserving full regression test runs for key milestones in the development lifecycle. This is workable if the limited tests produce a similar result in the short term. In this paper, we present a new approach for test suite selection that focuses on improving test effectiveness. The methodology described produces a pruned list of test cases required to test an application. The method has three components, the predictive component which makes use of statistical data, coverage based method digs the delta from the code to produce a pruned list of test cases, and decision based technique that prioritizes important test cases. Our experiments show that our approach results in a better utilization of compute resources and also decreases validation cycle thus reducing time to market. General Terms Testcase prioritization and selection